itsCausal: Interrupted Time Series in real world data

The User’s Guide

Aurélien Sallin, Daniel Ammann, Tobias Müller

2024-05-11

Why we need interrupted time series: the “no control group” scenario

  • Problem: In many real-world scenarios, finding a credible control group to evaluate a policy is not feasible.
  • Example:. Health care policies often affect all providers simultaneously, like changes in physician compensation or changes in guidelines.

  • Challenge: Standard interrupted time-series (ITS) may produce biased estimates because of poor prediction, poor forecasting, and simplification assumptions with panel data.

The use of ITS in the evaluation of health system interventions has increased considerably

“The risk of bias for ITS studies was high for 53.3% and very high for 19.2%”. (Hategeka et al. (2020), N = 120, 1990-2020)

  • The intervention was dependent of other changes.
  • Time series techniques were not propertly used.
  • Trends and time-dependent covariates were not correctly specified.
  • …

With “itsCausal”, we aim to deliver a set of recommendations for practitioners

  • Key Idea:
    • Provide guidance on the application of Interrupted Time Series
    • Enhance the traditional model with a machine learning-based method to address the “no control group” scenario and improve counterfactual forecasting.
    • Tailor the method to panel data where the number of units exceeds the number of time periods.
  • How it works:
    • Use machine learners and ensemble methods to train models on pre-intervention data.
    • Forecast onto the post-intervention period (“counterfactual”).
    • Compare actual outcomes with predicted counterfactuals to estimate causal effects.
    • Aggregate results for the desired population and the desired time period.

itsCausal was born from the practical need for effective healthcare monitoring

  • Monitor the effects of population-level interventions using rich (panel) claims data.
  • Example: two population-level interventions against low-value care (Vit. D testing) in Switzerland.

Development of a user’s guide for itsCausal

  • ML learners (random forest, gradient boosting, neural networks, catboost, lstm) with hyperparameters tuning.
  • Rolling-window approach for post-intervention forecasts with time-variant and invariant predictors.
  • Effect computed as the difference between the observed and the forecasted values.
  • Simulations of different data-generating processes show good performance.

Simulation of an ARMA Process

  • We benchmark our method with experimental evidence

    • Randomized controlled trial: Primary Care Physicians in Switzerland were sent an information letter combining professional norms and peer comparison feedback about vitamin D testing (Müller, Van Gestel, and Gerfin (2023))
    • ATE from itsCausal is within the 95% CI of the ATE from the RCT

  • We offer researchers and industry analysts in Public Health, Health Economics, and Health Services Research recommendations for the implementation of interrupted time-series.
  • We build tests and graphical representations of tests for the assumption of causal effects, especially for the “no-confounding” assumption.
  • We implement ML learners and provide guidance on how to implement them.

Conclusion

  • itsCausal: A powerful tool for estimating causal effects without a control group.
  • Benchmarking of the method with published research and experimental evidence.
  • User’s guide for sound implementation of ITS in real-world evidence, public health, health services research, and health economics.




🔗 itsCausal on GitHub
🌐 www.aureliensallin.ch

Identifying assumptions for the causal effect in its

The following assumptions must hold (see Cerqua, Letta, and Menchetti (2024)):

  1. There are no hidden forms of treatment leading to different potential outcomes (weak SUTVA).
  2. Additivity
  3. No anticipation and no confounding - Absence of anticipatory effects of the intervention on the covariates and the potential outcomes - Future covariates do not affect current potential outcomes - Covariates remain unaffected by the policy in the post-intervention period (post-treatment exogeneity of the covariates)
  4. Dynamic potential outcomes model: the potential outcomes absent the policy (the “counterfactual”) can be predicted using lagged values of the outcome and of the covariates.
  5. Post-intervention non-linear multi-step-ahead model: the counterfactual can be predicted for multiple periods ahead using lagged values of the outcomes until the intervention, conditional expectations of the outcome after the intervention, and the covariates.

References

Bernal, J Lopez, S Soumerai, and Antonio Gasparrini. 2018. “A Methodological Framework for Model Selection in Interrupted Time Series Studies.” Journal of Clinical Epidemiology 103: 82–91.
Brodersen, Kay H, Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L Scott. 2015. “Inferring Causal Impact Using Bayesian Structural Time-Series Models.”
Cerqua, Augusto, Marco Letta, and Fiammetta Menchetti. 2024. “Causal Inference and Policy Evaluation Without a Control Group.” arXiv Preprint arXiv:2312.05858.
Chernozhukov, Victor, Kaspar Wüthrich, and Yinchu Zhu. 2021. “An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls.” Journal of the American Statistical Association 116 (536): 1849–64.
Hategeka, Celestin, Hinda Ruton, Mohammad Karamouzian, Larry D Lynd, and Michael R Law. 2020. “Use of Interrupted Time Series Methods in the Evaluation of Health System Quality Improvement Interventions: A Methodological Systematic Review.” BMJ Global Health 5 (10): e003567.
Lopez Bernal, James, Steven Cummins, and Antonio Gasparrini. 2018. “The Use of Controls in Interrupted Time Series Studies of Public Health Interventions.” International Journal of Epidemiology 47 (6): 2082–93.
Müller, Tobias, Raf Van Gestel, and Michael Gerfin. 2023. “Pairing Professional Norms and Peer Comparison Feedback to Reduce Low-Value Care: Evidence from a Randomized Controlled Trial Among Primary Care Physicians.” Working Paper.
Turner, Simon L, Amalia Karahalios, Andrew B Forbes, Monica Taljaard, Jeremy M Grimshaw, and Joanne E McKenzie. 2021. “Comparison of Six Statistical Methods for Interrupted Time Series Studies: Empirical Evaluation of 190 Published Series.” BMC Medical Research Methodology 21 (1): 134.